NetworkTransform + CharacterController has "spooky behaviour" for other player's characters

Hello, I've got a question/bug. I've got a character using a NetworkTransform and a CharacterController. The velocity of the CharacterController drives parameters on the Animator. In the video the left side is the Host/Server, and the right is a Client. The log in the lower right shows the velocity read from the CharacterController of the client's player from the the host's perspective (if that makes sense). Sometimes the velocity of the client's character reads incorrectly, even though it isn't moving. This is causing a running-in-place behaviour you can see around halfway through the video. This running-in-place can be canceled by providing any amount of rotation with the mouse, which seems to update/correct the bad values.

Here's the related code for the players:

Code (CSharp):

1. protected void Update()
2. {
3.    if (isLocalPlayer)
4.     {
5.         // Movement
6.         Vector3 input = new Vector3(Input.GetAxis("Horizontal"), 0f, Input.GetAxis("Vertical"));
7.         characterController.SimpleMove(input.normalized * speed);
8.  
9.         // Rotation
10.         Ray mouse = Camera.main.ScreenPointToRay(Input.mousePosition);
11.         Plane plane = new Plane(Vector3.up, transform.position);
12.         float distance;
13.         plane.Raycast(mouse, out distance);
14.         transform.LookAt(mouse.GetPoint(distance));
15.     }
16.    
17.     // Smoothing using Lerp
18.     velocity = Vector3.Lerp(velocity, transform.InverseTransformDirection(characterController.velocity), 10f * Time.deltaTime);
19.     animator.SetFloat("Horizontal", velocity.x);
20.     animator.SetFloat("Vertical", velocity.z);
21. }

And the setup on the characters:



I'd rather not use NetworkAnimator because it doesn't smooth float parameters, which makes my blend tree chunky. I also don't particularly want to trust my clients like that.

Using NetworkTransform.targetSyncVelocity always reads Vector3.zero and calculating my own velocity from NetworkTransform.targetSyncPosition (syncPosition - transform.position) seems to be incorrect.