Cost/Disadvantage of using colliders for all animated fingers?

I am creating an avatar body for my VR player and I would like it to be able to bump into/knock over certain objects in my scene.

My avatar has a basic ragdoll collider setup. And I don't mind little gaps in places like the chest, elbow, knees...

But a single capsule collider for the entire hand is not very convincing.

I was wondering if anyone has tried adding colliders to each finger joint of tracked and animated hands. And if so was it disadvantageous or cause visual problems in any way?

I only intend on doing this to the local player's hands, and none of the other characters in my scene.