Cloth Component collision issue when set vertices to 0

I've got this coat with these buttons, the buttons are not separated models, it is attached to the coat, then when I set vertices to be 0.2, to give a cloth effect, these buttons start to falling off, then when I set vertices to be 0 in the buttons, they don't fall off, they get stuck in there, but then I get a problem with the collisions, like in the picture below :

.

PS : I want gravity to be enabled.

The collision is trying to make the cloth move away, but the vertices set to 0, does not move at all even with collisions.

Anyone does know how to solve that problem?, find a way to ignore collision when vertices are set to 0.
I have tried many things and did not work well, keep having the same issue.