Parented Object Scales Itself Absurdly

I did an object pick up system with raycast, created a empty game object as pickup point on character's model's hand (model is as a whole so I parented the pickup point on model's armature hand part).

When I click on interact button (Right Mouse Button) it picks up an object (assigns object's transform postion to hand pick up point's transform position and assigns itself as child to the pickup point) and when I click interact button the game object gets dropped (unparents itself).

The system works so far but has a bug: when I drop the object, the object's scale changes weirdly. It is more obvious when the character is walking. When the character stops and drops the object, sometimes it keeps its original scale and sometimes makes very very small change on scale.

What am I doing wrong?