Skinned Mesh Renderer quality

I'm trying to figure out the best way to optimize skinned characters and after reading the docs on the Skinned Mesh Renderer (https://docs.unity3d.com/Manual/class-SkinnedMeshRenderer.html) I have a quick question.

I was always under the impression that the more bones that influence a vert, the slower it is to calculate the deformation so keep as many verts down to 1 or 2 influences as possible.

However, now I've re-read the docs, it seems the quality setting sets the performance cost, not necessarily the actual number of influences per vert.

By this I mean, if a skinned mesh has all influences set to 1 bone, except 1 vert that has 4 bones, its quality setting needs to be 4 bones for that single vert. Does that slow all the other verts down as well, in which case they may as well all have 4 bones influence them, or is it still better to keep as few influences as possible even with the quality set to 4?

If it makes a difference, I'm looking at a quest VR game.

 

For my experience unless you have a huge amount of characters it doesn't make sense. Even the cheapest of mobiles phones will barely see any difference, I guess with PC is even more of a no issue.
I don't know how Unity handles it internally, but I think this set only the maximum of influences the skin can use, and won't use 4 for each skin even when not present.
If you set 1 bone on a 4 bone influences skin, Unity will simply ignore the remaining three making the animation looks strange.