Layermask in Unity

So last night I fought with the Layersystem in Unity.
Since I could´nt find a good explanation online, I want to share my findings here, since I don´t know where it belongs.

Without further ado:

Layer in Unity

The LayerMask is in the end an 32 bit-integer, which get´s used in 2 different ways:

variant A: the value of the int is the layernumber. aka 3 => Layer 3
variant B: the value of the int is meaningless, the int is used as bitmask for Raycasts etc. aka: 8 (...001000 binary) => only cast against Layer 3

Reading / assigning a Layermask to a GameObject:
since a GameObject can only be part of a single Layer the integer is supposed to have values no bigger than 32
aka. I want to assign a GameObject to Layer 3 I write:
gameObject.layer = 3;
=> variant A is used.

Doing Raycasts etc:
since we want to specify which Layers are ignored, and which are not, we are going to use a bitmask
aka. I want to cast only against Layer 3 I write:
int layer = 1 << 3 = 8 (...001000 binary) => using bitshifts to generate the bitmask.
Physics.Raycast(origin, direction, distance, *layer*)

When do I have variant A and when variant B?

exposing a Layermask variable to the inspector and have it´s value assigned that way: variant B
using LayerMask.NameToLayer : variant A

My own opinion:
since bitmasks and the unclear nature, when which type is needed and generated are not that welcoming to new Developers.
I would like to seperate the LayerMask (variant B) and The LayerIndex (varaint A) into 2 seperate types.
These should be convertible into each other:

i.e:
public LayerIndex[] LayerMaskToLayerIndex(LayerMask mask);
public LayerMask LayerIndexToLayerMask(params LayerIndex[] layers);

I spent far to much time figuring this out, an I hope I helped at least one other person.

Grettings
Grizzly

 

Although I agree that layers and layermasks can be confusing initially (or even semi-permanently), I think some of your reaction is due to a minor lack of rigor in your naming and understanding. What jumps out at me from your post is the statement: "Reading / assigning a Layermask to a GameObject". That's not a reasonable thing to say. In Unity, you read or assign a GameObject's layer, not its layer mask.

The two initial "variants" you describe are also a bit wrong due to this mix up. You stated that both A and B are descriptions of a LayerMask. But "A" is a description of a GameObject's layer, not its layer mask.

Put simply, GameObjects are always assigned to one and only one layer at a time. The layer is an integer property, and the values will range from 0 to 31. For convenience, Unity allows us to assign names to these integer values via the Layers dialog. But ultimately the layers are integers. A layer is simply an integer value assignment on a game object ranging from 0 to 31. It controls various rendering and physics behaviors.

A LayerMask is not the same thing as a Layer. As you've stated, a LayerMask is a bitmask that is generated from one or more layer values. A LayerMask can be used for various things, but developers often use it most for filtering interactions in Raycast (and similar) commands. I think it's important to restate that a Game Object doesn't "have" a layer mask. You can create a layer mask that includes a game object's layer, such that the game object would be found via a Raycast. But it's kind of like claiming that if an object is contained in some list, then the object has that list as its property, which isn't the case.

What I do agree with is that there are times when it's confusing whether I should be using a Layer, or a LayerMask. Raycast is relatively simple, since the expected parameter is named layerMask. But I've definitely had bugs where I thought I needed to pass in a layermask and I passed in the layer instead, or vice versa.

Anyway, I'd say you're probably just started to get the hang of this, and it'll become pretty clear and straightforward soon. But I don't think it quite means that underlying changes to Unity should be made as you described. But it's good to write up thoughts like that. Personally, doing so helps me to cement my understanding.