Multiple fixed joints breaking the weight of the object

On a part of my project I am working on, which is a VR game, I have it so you can use both hands to grab an object using a fixed joint. It works fine, except if you grab it with two hands, the item becomes weightless, if you let go with the second hand, the weight is restored, but if you let go with the first hand, it remains weightless until you let go again. It is kinda similar to if I had just parented the object to the hand. Is there something I am doing wrong, or do I have to create a work around?

The code basically creates a joint on the held item and one onto the VR hand, then sets the connected body to each other's rigidbody, along with disabling preprocessing and collisions.

 

Check the mass scale. Make sure it's always 1 to 1.
I had that issue too, even with good mass scale.
I accidentialy fixed it, when started using Hinge Joint. I wanted to make object in hand to rotate slightly while being held.
Also make sure you are moving your hand considering its weight too. I'm using Impulse force mode for that.