Question Optimal Character Rig For Animation Performance?

I'm trying to make a VR game with Muffin and Donut characters and I want to rig and animate them in an optimal way for performance. Performance is important because there could be a lot of characters in a scene, up to 50. And I might want this VR game to run on Meta Quest hardware, if that's even possible. The characters are designed to be simple and don’t require deformations.

I heard that animation on whole objects is more performant in Unity than a skinned mesh character rig. Since I don’t need deformations, I was initially thinking I would build character rigs without joints, and the rig controls would just control body part objects directly via constraints. For example, the left arm object is parent constrained to the left arm control, which itself is aim constrained to the left shoulder control (no joints involved). Then, after an animation is complete, that animation will be baked onto the objects and exported to Unity.

My question is, I’m I avoiding the use of a skinned mesh rig for negligible performance benefits, if any? For example, if I use a skinned mesh rig, and each vertex is only controlled by one joint, would that not be very similar performance-wise to animating the objects directly?

I included reference videos of the characters I plan to rig soon.