[Help] Syncing pushing objects in world

Hello, I am developing a game where the players can push an object like a Cube on runtime. However I am having some issues with keeping it smooth for the people who are not the host.

I have developed a few solutions but they still don't feel up to scratch.
It makes me wonder how games like Human Fall Flat and Gang Beasts managed to pull off so much physics and making it so smooth

Does anyone have any ideas?

Currently the client sends input to the server and predicts his movement but for objects which are push able it doesn't work great and is just flickering [If that makes sense]

Note here's a good GDC talk about network physics: https://www.gdcvault.com/play/1022195/Physics-for-Game-Programmers-Networking

Side note: I'm not using Unet, I'm using my own custom net code