Rigidbody not responding to changes properly

Hey, a problem popped up. I have two rigidbody objects in my scene and the other one seems to not respond to the changes in the scene all that well. There's a cube the player can move and rotate with the mouse and the rigidbody is set over it, but when you move the cube, the rigidbody is stuck in air for some reason. It takes a while and then it might fall down, but the response is never immediate.

I wonder what the problem is. I have the collision detection set to continuous dynamic and have a function to awake the rigidbody up, but nothing seems to help. The rigidbody drops and collides normally with the cube at the beginning of the scene. There's no other scripts on the rigidbody other than the simple WakeUp function and an order to use ContinuousDynamic collision detection mode in FixedUpdate function.

Do you guys have any tips on how to solve this?