Sync Network Movement And Physics

Hello... first of all let me explain what I am trying to accomplish here.

I am trying to create a simple game. There's a field, 2 goals, 2 players, and 1 ball (square ball) in the middle. The players can only move within their half of the field and can press Z to hit the ball and they can also simply bump into it to move it like a regular rigidbody colliision.

I have accomplish so far a (not very well done because I am new at this stuff) way to sync the players' movement.
* Player 1 does input.
* Input is processed locally and sent to server.
* Server sends input to second player. (Only processed 100ms later based on timestamp from Player 1)
* 100ms later server processes the input and sends to both player.
* Both players re-adjust the position of Player 1 if there was an error. (Necessary to re-simulate the previous inputs, I got it working good I'd say, but it's not perfect, it snaps players sometimes.)
Everything is ran with FixedUpdate and each input done is locked to run for 3 FixedUpdates.

Now, I cannot understand or figure out how to sync the ball for both players. If the ball has to be controlled by the server, how can both players see it in real time if the local player sees his character in real time and the other character with a 100ms lag?

By the way, I tried doing this using Network Transform. It didn't work well at all.

I appreciate any help