Question NavAgent doesn't work?

Hi all, I'm trying to make a simple click to move system in my game. Still a bit new to NetCode so I am unfamiliar with the way all the syntax works. I have done a bunch of debugging and I have narrowed it down but not sure what the problem is. I am sending the data properly, and its being received properly, but the player just doesn't move. No idea what is going on.

Code (csharp):

1.  
2. [ServerRpc]
3.     private void SetTargetServerRpc(Vector3 _loc)
4.     {
5.         agent.SetDestination(_loc);
6.     }

Here is the code that is being called after I click a spot. When I debug it, the click location is being sent, and the player has a NavMeshAgent, so I have no idea why it wouldn't be moving. I even tried running the server on my Unity editor so I can check what is going on, and the player shows the new destination being set properly on the server side, but there is no movement.

Any ideas?