Multiplayer environment only works correctly for host and not for the clients

Hi, I created a multiplayer environment with Photon fusion with cinemachine to enable 3rd person camera. When I run the application, host movements are working fine but clients' movements are not going in the desired direction. Sometimes there exist frames drop when trying to move clients. But nothing happened to the host. The code was pretty much similar to the code in photon fusion main documentation(only difference is the 3rd person camera with a human model). Any idea what would be the reason for this?(movement was done using keys+mouse rotation)

Note: FixedUpdateNetwork looks like this,

Code (CSharp):

1.  public override void FixedUpdateNetwork()
2.     {
3.         //Get the input from the network
4.         if (GetInput(out NetworkInputData networkInputData) && Runner.IsForward)
5.         {
6.             Vector3 direction = new Vector3(networkInputData.movementInput.x, 0f, networkInputData.movementInput.y).normalized;
7.             if (direction.magnitude >= 0.1f)
8.             {
9.                 animator.SetBool("Walk", true);
10.                 animator.SetBool("Idle", false);
11.  
12.                 //rotation
13.                 float targetAngle = Mathf.Atan2(direction.x, direction.z) * Mathf.Rad2Deg + playerCamera.eulerAngles.y;
14.                 float angle = Mathf.SmoothDampAngle(transform.eulerAngles.y, targetAngle, ref turnCalmVelocity, turnClamTime);
15.                 transform.rotation = Quaternion.Euler(0f, angle, 0f);
16.  
17.                 Vector3 moveDirection = Quaternion.Euler(0f, targetAngle, 0f) * Vector3.forward;
18.  
19.                 networkCharacterControllerPrototypeCustom.Move(moveDirection.normalized * playerSpeed * Time.deltaTime);
20.  
21.             }
22.             else
23.             {
24.                 animator.SetBool("Idle", true);
25.                 animator.SetBool("Walk", false);
26.             }
27.         }
28.     }