FreeLook Camera Jittering When Rotating Camera with Mouse

I'm on unity 2019.1.12f1. CM version 2.3.4.

My CineBrain is set to SmartUpdate. I have no physics objects. Every frame on Update I do re-position the camera's LookAt transforms position. I'm doing that with the following code:

Vector3 dir = heroPlayer.position - cam_main.transform.position; //main direction
dir = dir + Vector3.up; //move up to eye level
dir = dir.normalized * 5f; //normalize and scale
cm_mainLookAtPoint.transform.position = heroPlayer.position + dir; //set position

On the last line- I've tried lerping the lookAt new position with a value of .9f on every frame, but that did not remove the jitter.. I tried changing my FreeLook StandbyUpdate from RoundRobin to Always, but that did not remove the jitter... My FPS is around 400-700 consistently (nothing crazy happening in my scene). In the editor the character is moving smoothly, but the camera is jittery.

I've also built the project and in the build there's also the same issue. It visually shows up less harsh (more like image ghosting), but indeed the same problem.

Any ideas what's going on here?

 

No problem.
Inspector: https://imgur.com/a/9WYj6BP <---Image link works
Hierarchy: https://imgur.com/a/x0W9nua <---Image link works

Regarding the project - it's main virtual camera is the CM-Main-FreeLook. When I hold down right click it changes priorities so that the camera goes into CM-TargetingFreeLook. Both have the same jitter issue, but even if I never change priorities and stay in the CM-Main-Freelook I see this jitter issue.

The CameraManager never moves or rotates. The Character Controller moves and the main virtual camera uses this as its Follow.

It is worth noting about 20% of the time when I play the project I don't see the jitter...The other 80% it appears... I have no idea why it would be appearing as the frame-rate is still always above 400fps...The character doesn't jitter in the editor view - only the camera.

 

Thanks for the reply Gregoryl. Really appreciate how active/helpful you are on these forums.

I switched my LookAt to be the same as the Follow (making it the Character Controller) and did the tracked object offset - and no longer see the jitter. The offset indeed helps to make sure it's pointed correctly closer to the characters head. Shouldn't the custom target that I have moving in code work also though? In general is this not recommended? Imagine I had a cube animating in the distance that I wanted to switch the character to looking at. Shouldn't it animate smoothly still?