Unable to interact with a network object as client

I'm trying to add velocity to a network object on the scene. I have the following script attached to the host and the client an I can add velocity to the ball with the host character, but I cannot add velocity to the ball object on the client. I am spawning the ball object like this in OnNetworkSpawn:

ball.GetComponent<NetworkObject>().Spawn();

I am not getting any error messages.

I have the network object, network transform and network rigidbody components attached to the ball object.

Code (CSharp):

1. [ServerRpc(RequireOwnership = false)]
2.     public void hitServerRpc()
3.     {    
4.         if (Input.GetKeyUp(KeyCode.Mouse0))
5.         {
6.             isCharging = false;
7.  
8.             if (ballIn == true)
9.             {              
10.                 rbBall.velocity = (cam.forward * hitForce);
11.             }
12.  
13.         }
14.      
15.     }
16.     private void OnTriggerEnter(Collider other)
17.     {
18.         if (other.gameObject.name == "Ball")
19.         {
20.             ballIn = true;
21.             rbBall = other.GetComponent<Rigidbody>();
22.         }
23.      
24.     }
25.     void OnTriggerExit(Collider other)
26.     {
27.         if (other.gameObject.name == "Ball")
28.         {
29.             ballIn = false;
30.         }
31.     }